When training complex neural networks, memory usage can be an important bottleneck. The question of when to rematerialize, i.e., to recompute intermediate values rather than retaining them in memory, becomes critical to achieving the best time and space efficiency. In this work we consider the rematerialization problem and devise efficient algorithms that use structural characterizations of computation graphs---treewidth and pathwidth---to obtain provably efficient rematerialization schedules. Our experiments demonstrate the performance of these algorithms on many common deep learning models.

Neural Information Processing Systems http://nips.cc/

Though the paper is clearly written and presents an interesting approach for large model training. It also allows a GPU to train models that cannot fit into the GPU on-device memory. However, it completely ignores how fast a model can be trained after rematerialization. Given that it already takes a significant amount of time to train a large model on a single GPU and there are many alternatives to train large models without sacrificing too much on computation, it does not seem like the paper provides strong enough evidence over choosing an alternative approach. The paper claims that the "feasibility" of training large models as the main motivation, but the memory itself is not the only bottleneck and the paper completely ignores the training speed side of the story.

When training complex neural networks, memory usage can be an important bottleneck. The question of when to rematerialize, i.e., to recompute intermediate values rather than retaining them in memory, becomes critical to achieving the best time and space efficiency. In this work we consider the rematerialization problem and devise efficient algorithms that use structural characterizations of computation graphs---treewidth and pathwidth---to obtain provably efficient rematerialization schedules. Our experiments demonstrate the performance of these algorithms on many common deep learning models.

When training complex neural networks, memory usage can be an important bottleneck. The question of when to rematerialize, i.e., to recompute intermediate values rather than retaining them in memory, becomes critical to achieving the best time and space efficiency. In this work we consider the rematerialization problem and devise efficient algorithms that use structural characterizations of computation graphs---treewidth and pathwidth---to obtain provably efficient rematerialization schedules. Our experiments demonstrate the performance of these algorithms on many common deep learning models. Papers published at the Neural Information Processing Systems Conference.